Augmented reality camera for use with 3d metrology equipment in forming 3d images from 2d camera images

ABSTRACT

A method uses a two-dimensional (2D) camera in two different positions to provide first and second 2D images having three common cardinal points. It further uses a three-dimensional (3D) measuring device to measure two 3D coordinates. The first and second 2D images and the two 3D coordinates are combined to obtain a scaled 3D image.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Ser.No. 62/055,030, filed Sep. 25, 2014, the entire disclosure of which isincorporated herein by reference.

BACKGROUND OF THE INVENTION

The present disclosure relates to augmented reality and moreparticularly to a stand-alone augmented reality camera that is utilizedto generate two-dimensional (2D) images that are formed intothree-dimensional (3D) images through use of scale measurements made by3D metrology equipment.

Augmented reality (AR) is a relatively new type of technology that grewout of virtual reality. Augmented reality merges, superimposes, ortransprojects actual real-world information or data with, on, into, oronto virtual information or data. That is, the virtual information ordata “augments,” compliments or supplements the actual sensed, measured,captured or imaged real-world information or data related to some objector scene to give the user an enhanced view or perception of the realworld object or scene. Augmented reality applications include technicalor industrial areas such as part, component or device manufacturing andassembly and/or repair and maintenance, and facility, building orstructure layout and construction. A number of modern-day ARapplications are disclosed athttp://en.wikipedia.org/wiki/Augmented_reality.

The actual information or data relating to the part, component ordevice, or area or scene, may be obtained in various ways using variousdevices. One type of device that may provide the actual information ordata includes a 3D metrology device, such as, for example, a coordinatemeasurement device in the nature of a portable articulated armcoordinate measurement machine (AACMM) or a laser tracker. This type ofmeasurement device may measure and provide the actual 3D coordinates ofthe part, component, device, area or scene in the nature of the threetranslational coordinates (e.g., the x, y and z or Cartesiancoordinates) as well as the three rotational coordinates (e.g., pitch,roll and yaw). As such, the measurement device may be understood asproviding for six degrees of freedom (i.e., six-DOF).

A camera may also be used to take still or video images of the actualpart, component or device, and/or a desired area or scene by itself orthat surrounding or associated with the part, component or device.

The virtual information or data may be stored artificial informationregarding the part, component, device, area or scene. The stored virtualinformation or data may be related to the design of the part, component,device, area or scene ranging from, for example, simple text or symbolsto relatively more complex, graphic 3D CAD design data. Besides visualinformation, the stored virtual information or data may also compriseaudible or sound information or data. The stored virtual information ordata may also relate to information such as textual or part, componentor device repair or maintenance instructions, or visual informationdepicting parts, components or devices that may be used, for example, inthe design of an office or manufacturing and/or repair facility (e.g., abuilding or facility layout).

The combined actual and virtual information or data in an AR system isusually digital in nature and may be delivered in real-time (i.e., asthe actual information is being measured or sensed) to a user on adisplay screen that may be in many different types or forms, such asthat associated with, for example, a desktop or laptop computer monitor,tablet, smartphone or even a head-mounted display such as thoseassociated with glasses, hats or helmets. Audio information may bedelivered through a speaker.

As mentioned, one type of 3D metrology or coordinate measurement deviceincludes a portable AACMM. These AACMMs have found widespread use in themanufacturing or production of parts where there is a need to rapidlyand accurately verify the dimensions of the part during various stagesof the manufacturing or production (e.g., machining). Portable AACMMsrepresent an improvement over known stationary or fixed, cost-intensiveand relatively difficult to use measurement installations, particularlyin the amount of time it takes to perform dimensional measurements ofrelatively complex parts. Typically, a user of a portable AACMM simplyguides a probe along the surface of the part or object to be measured.The measurement data are then recorded and provided to the user. In somecases, the data are provided to the user in visual form, for example, in3D form on a computer screen. In other cases, the data are provided tothe user in numeric form, for example when measuring the diameter of ahole, the text “Diameter=1.0034” is displayed on a computer screen.

An example of a prior art portable AACMM is disclosed in U.S. Pat. No.5,402,582 ('582) to Raab, the contents of which are incorporated hereinby reference. The '582 patent discloses a 3D measurement systemcomprised of a manually-operated AACMM having a support base on one endand a measurement probe at the other end. Also, U.S. Pat. No. 5,611,147('147) to Raab, the contents of which are incorporated herein byreference, discloses a similar AACMM. In the '147 patent, the AACMMincludes a number of features including an additional rotational axis atthe probe end, thereby providing for an arm with either a two-two-two ora two-two-three axis configuration (the latter case being a seven axisarm).

Another type of 3D metrology or coordinate measurement device belongs toa class of instruments known as a laser tracker that measures the 3Dcoordinates of a point by sending a laser beam to the point. The laserbeam may impinge directly on the point or on a retroreflector target incontact with the point. In either case, the laser tracker instrumentdetermines the coordinates of the point by measuring the distance andthe two angles to the target. The distance is measured with a distancemeasuring device such as an absolute distance meter or aninterferometer. The angles are measured with an angle measuring devicesuch as an angular encoder. A gimbaled beam-steering mechanism withinthe instrument directs the laser beam to the point of interest.

The laser tracker is a particular type of coordinate measurement devicethat tracks the retroreflector target with one or more laser beams itemits. The laser tracker is thus a “time-of-flight” (TOF) type ofmeasurement device. Coordinate measurement devices closely related tothe laser tracker are the laser scanner and the total station. The laserscanner steps one or more laser beams to points on a surface of anobject. It picks up light scattered from the surface and from this lightdetermines the distance and two angles to each point. The total station,which is most often used in surveying applications, may be used tomeasure the coordinates of diffusely scattering or retroreflectivetargets.

Ordinarily the laser tracker sends a laser beam to a retroreflectortarget. A common type of retroreflector target is the sphericallymounted retroreflector (SMR), which comprises a cube-cornerretroreflector embedded within a metal sphere. The cube-cornerretroreflector comprises three mutually perpendicular mirrors. Thevertex, which is the common point of intersection of the three mirrors,is located at the center of the sphere. Because of this placement of thecube corner within the sphere, the perpendicular distance from thevertex to any surface on which the SMR rests remains constant, even asthe SMR is rotated. Consequently, the laser tracker can measure the 3Dcoordinates of the object surface by following the position of an SMR asit is moved over the surface. Stating this another way, the lasertracker needs to measure only three degrees of freedom (one radialdistance and two angles) to fully characterize the 3D coordinates of asurface.

One type of laser tracker contains only an interferometer (IFM) withoutan absolute distance meter (ADM). If an object blocks the path of thelaser beam from one of these trackers, the IFM loses its distancereference. The operator must then track the retroreflector to a knownlocation to reset to a reference distance before continuing themeasurement. A way around this limitation is to put an ADM in thetracker. The ADM can measure distance in a point-and-shoot manner, asdescribed in more detail below. Some laser trackers contain only an ADMwithout an interferometer. U.S. Pat. No. 7,352,446 ('446) to Bridges etal., the contents of which are incorporated herein by reference,describes a laser tracker having only an ADM (and no IFM) that is ableto accurately scan a moving target. Prior to the '446 patent, absolutedistance meters were too slow to accurately find the position of amoving target.

A gimbal mechanism within the laser tracker may be used to direct alaser beam from the tracker to the SMR. Part of the light retroreflectedby the SMR enters the laser tracker and passes onto a position detector.A control system within the laser tracker can use the position of thelight on the position detector to adjust the rotation angles of themechanical axes of the laser tracker to keep the laser beam centered onthe SMR. In this way, the tracker is able to follow (track) an SMR thatis moved over the surface of an object of interest.

Angle measuring devices such as angular encoders are attached to themechanical axes of the tracker. The one distance measurement and twoangle measurements performed by the laser tracker are sufficient tocompletely specify the three-dimensional location of the SMR at anypoint on the surface of the object being measured.

Several laser trackers have been disclosed for measuring six, ratherthan the ordinary three, degrees of freedom. These six degrees offreedom include three translational degrees of freedom and threeorientational degrees of freedom, as described in more detailhereinafter. Exemplary six degree-of-freedom (six-DOF or 6-DOF) lasertracker systems are described by U.S. Pat. No. 7,800,758 ('758) toBridges et al., U.S. Pat. No. 8,525,983 ('983) to Bridges et al., andU.S. Pat. No. 8,467,072 ('072) to Cramer et al., the contents of each ofwhich are incorporated herein by reference.

These six-DOF laser trackers may include a separate probe having aretroreflector for which the laser tracker measures the six degrees offreedom. The six degrees of freedom of the probe measured by the lasertracker may be considered to include three translational degrees offreedom and three orientational degrees of freedom. The threetranslational degrees of freedom may include a radial distancemeasurement between the laser tracker and the retroreflector, a firstangular measurement, and a second angular measurement. The radialdistance measurement may be made with an IFM or an ADM within the lasertracker. The first angular measurement may be made with an azimuthangular measurement device, such as an azimuth angular encoder, and thesecond angular measurement made with a zenith angular measurementdevice, such as a zenith angular encoder. Alternatively, the firstangular measurement device may be the zenith angular measurement deviceand the second angular measurement device may be the azimuth angularmeasurement device. The radial distance, first angular measurement, andsecond angular measurement constitute three coordinates in a sphericalcoordinate system, which can be transformed into three coordinates in aCartesian coordinate system or another coordinate system.

The three orientational degrees of freedom of the probe may bedetermined using a patterned cube corner, as described in theaforementioned patent '758. Alternatively, other methods of determiningthe three orientational degrees of freedom of the probe may be used. Thethree translational degrees of freedom and the three orientationaldegrees of freedom fully define the position and orientation of thesix-DOF probe (and, thus, of the probe tip) in space. It is important tonote that this is the case for the systems considered here because it ispossible to have systems in which the six degrees of freedom are notindependent so that six degrees of freedom are not sufficient to fullydefine the position and orientation of a device in space. The term“translational set” is a shorthand notation for three degrees oftranslational freedom of a six-DOF accessory (such as the six-DOF probe)in the laser tracker frame of reference. The term “orientational set” isa shorthand notation for three orientational degrees of freedom of asix-DOF accessory (e.g., the probe) in the laser tracker frame ofreference. The term “surface set” is a shorthand notation forthree-dimensional coordinates of a point on the object surface in thelaser tracker frame of reference as measured by the probe tip.

Other known types of 3D metrology devices include the aforementioned TOFlaser scanners and also triangulation scanners.

While some innovations have already been made in the area of augmentedreality for use with various types of 3D metrology devices, there is aneed for novel applications of augmented reality for use with 3Dmetrology devices such as AACMMs, laser trackers, TOF laser scanners,and triangulation scanners.

BRIEF DESCRIPTION OF THE INVENTION

In accordance with an embodiment of the present invention, a methodcombines at least two two-dimensional (2D) images into athree-dimensional (3D) image, the method including steps of providing anaugmented reality (AR) camera; providing a 3D metrology instrumenthaving a frame of reference, the AR camera being separate from the 3Dmetrology instrument, wherein the 3D metrology instrument is selectedfrom the group comprising a laser tracker, a time-of-flight (TOF) laserscanner, and an articulated arm coordinate measurement machine; in afirst instance, forming a first 2D image with the AR camera in a firstlocation; in a second instance, moving the AR camera to a secondlocation that is different from the first location, and forming a second2D image with the AR camera; finding at least three cardinal points incommon to both the first and second 2D images; registering the first andsecond 2D images to obtain an unsealed composite 3D image based at leastin part on the at least three found cardinal points; identifying a firstselected cardinal point and a second selected cardinal point from amongthe at least three found cardinal points; measuring 3D coordinates of afirst reference point and a second reference point with the 3D metrologyinstrument, the first reference point and the second reference pointbeing within a region covered by both the first 2D image and the second2D image; determining 3D coordinates of each of the first and secondselected cardinal points based at least in part on the measured 3Dcoordinates of the first reference point and the second reference point;and creating the 3D image as a scaled composite 3D image from thedetermined 3D coordinates of each of the first and second selectedcardinal points, and from the unsealed composite 3D image.

These and other advantages and features will become more apparent fromthe following description taken in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWING

The subject matter, which is regarded as the invention, is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The foregoing and other features, and advantages ofthe invention are apparent from the following detailed description takenin conjunction with the accompanying drawings in which:

FIG. 1 is a perspective view of a portable articulated arm coordinatemeasurement machine (AACMM) in accordance with an embodiment of theinvention;

FIG. 2 is a perspective view of a laser tracker in accordance with anembodiment of the invention;

FIG. 3 is a perspective view of a time-of-flight (TOF) laser scanner inaccordance with an embodiment of the invention;

FIG. 4 is a perspective view of a triangulation scanner in accordancewith an embodiment of the invention;

FIG. 5 illustrates an arrangement of an augmented reality camera and a3D metrology device used for taking 2D images of a surface of an objectand forming 3D images from the 2D images; and

FIG. 6 is a flowchart of steps in a method according to embodiments ofthe present invention for taking 2D images of a surface of an object andforming 3D images from the 2D images.

The detailed description explains embodiments of the invention, togetherwith advantages and features, by way of example with reference to thedrawings.

DETAILED DESCRIPTION OF THE INVENTION

FIGS. 1-4 illustrate exemplary three-dimensional (3D) metrology devicesor instruments, according to various embodiments of the presentinvention. These devices include an articulated arm coordinatemeasurement machine (AACMM) 100, a laser tracker 200, a time-of-flight(TOF) laser scanner 300, and a triangulation scanner 400, collectivelyreferred to herein as 3D instruments. It should be appreciated thatwhile embodiments herein may refer to specific 3D instruments, theclaimed invention should not be so limited. Also, the variousembodiments may be used alternatively in other 3D instruments ormetrology devices, such as but not limited to laser line probes, totalstations and theodolites for example.

Referring to FIG. 1, an AACMM 100 is illustrated according to variousembodiments of the present invention. An articulated arm is one type ofcoordinate measurement machine. The AACMM 100, which may be portable,may be the same as, or similar to, the AACMM described in U.S. Pat. No.8,533,967 ('967) to Bailey et al., the contents of which areincorporated herein by reference. The exemplary AACMM 100 may comprise asix or seven axis articulated coordinate measurement device having aprobe end 401 that includes a measurement probe housing 102 coupled toan arm portion 104 of the AACMM 100 at one end.

The arm portion 104 comprises a first arm segment 106 coupled to asecond arm segment 108 by a rotational connection having a firstgrouping of bearing cartridges 110 (e.g., two bearing cartridges). Asecond grouping of bearing cartridges 112 (e.g., two bearing cartridges)couples the second arm segment 108 to the measurement probe housing 102.A third grouping of bearing cartridges 114 (e.g., three bearingcartridges) couples the first arm segment 106 to a base 116 located atthe other end of the arm portion 104 of the AACMM 100. Each grouping ofbearing cartridges 110, 112, 114 provides for multiple axes ofarticulated movement. Also, the probe end 401 may include a measurementprobe housing 102 that comprises the shaft of the seventh axis portionof the AACMM 100 (e.g., a cartridge containing an encoder system thatdetermines movement of the measurement device, for example a contactprobe 118, in the seventh axis of the AACMM 100). In this embodiment,the probe end 401 may rotate about an axis extending through the centerof measurement probe housing 102. In use the base 116 is typicallyaffixed to a work surface.

Each bearing cartridge within each bearing cartridge grouping 110, 112,114 typically contains an encoder system (e.g., an optical angularencoder system). The encoder system (i.e., transducer) provides anindication of the position of the respective arm segments 106, 108 andcorresponding bearing cartridge groupings 110, 112, 114 that alltogether provide an indication of the position of the probe 118 withrespect to the base 116 (and, thus, the position of the object beingmeasured by the AACMM 100 in a certain frame of reference—for example alocal or global frame of reference).

The probe 118 is detachably mounted to the measurement probe housing102, which is connected to bearing cartridge grouping 112. A handleaccessory 126 may be removable with respect to the measurement probehousing 102 by way of, for example, a quick-connect interface. Inexemplary embodiments, the probe housing 102 houses a removable probe118, which is a contacting measurement device and may have differenttips 118 that physically contact the object to be measured, including,but not limited to: ball, touch-sensitive, curved and extension typeprobes. In other embodiments, the measurement is performed, for example,by a non-contacting device such as a laser line probe (LLP). In anembodiment, the handle 126 is replaced with the LLP using thequick-connect interface. Other types of accessory devices may replacethe removable handle 126 to provide additional functionality. Examplesof such accessory devices include, but are not limited to, one or moreillumination lights, a temperature sensor, a thermal scanner, a bar codescanner, a projector, a paint sprayer, a camera, a video camera, anaudio recording system or the like, for example.

In accordance with an embodiment, the base 116 of the portable AACMM 100contains or houses an electronic data processing system 130 thatincludes a base processing system that processes the data from thevarious encoder systems within the AACMM 100 as well as datarepresenting other arm parameters to support 3D positional calculations,and resident application software that allows for relatively completemetrology functions to be implemented within the AACMM 100.

As discussed in more detail below, the electronic data processing system130 in the base 116 may communicate with the encoder systems, sensors,and other peripheral hardware located away from the base 116 (e.g., aLLP that can be mounted to or within the removable handle 126 on theAACMM 100). The electronics that support these peripheral hardwaredevices or features may be located in each of the bearing cartridgegroupings 110, 112, 114 located within the portable AACMM 100.

Referring to FIG. 2, there illustrated is an exemplary laser trackersystem 200 that includes a laser tracker 202, a retroreflector target204, an electronic data processing system 206, and an optional auxiliarycomputer 208. The laser tracker 200 may be similar to that described incommonly owned U.S. Provisional Application Ser. No. 61/842,572, filedon Jul. 3, 2013, the contents of which are incorporated herein byreference. It should be appreciated that while the electronic dataprocessing system 206 is illustrated external to the laser tracker 200,this is for exemplary purposes and the electronic data processing system206 may be arranged within the housing of the laser tracker 200. Anexemplary gimbaled beam-steering mechanism 210 of laser tracker 200comprises a zenith carriage 212 mounted on an azimuth base 214 androtated about an azimuth axis 216. A payload 218 is mounted on thezenith carriage 212 and rotated about a zenith axis 220. Zenith axis 220and azimuth axis 216 intersect orthogonally, internally to tracker 200,at gimbal point 222, which is typically the origin of the localcoordinate system frame of reference for distance measurements.

A laser beam 224 virtually passes through the gimbal point 222 and ispointed orthogonal to zenith axis 220. In other words, laser beam 224lies in a plane approximately perpendicular to the zenith axis 220 andthat passes through the azimuth axis 216. Outgoing laser beam 224 ispointed in the desired direction by rotation of payload 218 about zenithaxis 220 and by rotation of zenith carriage 212 about azimuth axis 216.A zenith angular encoder 226, internal to the tracker 220, is attachedto a zenith mechanical axis aligned to the zenith axis 220. An azimuthangular encoder 228, internal to the tracker, is attached to an azimuthmechanical axis aligned to the azimuth axis 216. The zenith and azimuthangular encoders 226, 228 measure the zenith and azimuth angles ofrotation to relatively high accuracy. Outgoing laser beam 224 travels tothe retroreflector target 204, which might be, for example, aspherically mounted retroreflector (SMR).

The distance to the retroreflector target 204 is determined by theelectronic data processing system 206 in response to a signal from ameasuring device, such as an absolute distance meter (ADM) or aninterferometer for example. By measuring the radial distance betweengimbal point 222 and retroreflector 204, the rotation angle about thezenith axis 220, and the rotation angle about the azimuth axis 216, theposition of retroreflector 204 and thus the 3D coordinates of the objectbeing inspected is found by the electronic data processing system 206within the local spherical coordinate system of the tracker.

Referring to FIG. 3, an exemplary laser scanner 300 is shown inaccordance with embodiments of the present invention. The laser scanner300 has a measuring head 302 and a base 304. The laser scanner 300 maybe similar to that described in U.S. Published Patent Application No.2014/0078519 to Steffey et al., the contents of which are incorporatedby reference herein. The measuring head 302 is mounted on the base 304such that the laser scanner 300 may be rotated about a vertical axis306. In one embodiment, the measuring head 302 includes a gimbal point308 that is a center of rotation about a vertical axis 306 and ahorizontal axis 310. In an embodiment, the measuring head 302 has arotary mirror 312, which may be rotated about the horizontal axis 310.The rotation about the vertical axis 306 may be about the center of thebase 304. In an embodiment, the vertical (azimuth) axis 306 and thehorizontal (zenith) axis 310 intersect at the gimbal point 308, whichmay be an origin of a coordinate system.

The measuring head 302 is further provided with an electromagneticradiation emitter, such as light emitter 314 for example, that emits anemitted light beam 316. In one embodiment, the emitted light beam 316 iscoherent light, such as a laser beam for example. The laser beam mayhave a wavelength range of approximately 300 to 1600 nanometers, forexample 790 nanometers, 905 nanometers, 1550 nm, or less than 400nanometers. It should be appreciated that other electromagneticradiation beams having greater or smaller wavelengths may also be used.The emitted light beam 316 may be amplitude or intensity modulated, forexample, with a sinusoidal waveform or with a rectangular waveform. Theemitted light beam 316 is emitted by the light emitter 314 onto therotary mirror 312, where it is deflected to the environment. A reflectedlight beam 318 is reflected from the environment by an object 320. Thereflected or scattered light is intercepted by the rotary mirror 312 anddirected into a light receiver 322. The directions of the emitted lightbeam 316 and the reflected light beam 318 result from the angularpositions of the rotary mirror 312 and the measuring head 302 about theaxis 306 and axis 310, respectively. These angular positions in turndepend on the rotary drives that cause rotations of the rotary mirror312 and the measuring head 302 about the axis 310 and axis 306,respectively. Each of the axes 310, 306 include at least one angulartransducer 324, 326 for measuring angle. The angular transducer may bean angular encoder.

Coupled to the light emitter 314 and the light receiver 322 is anelectronic data processing system 328. The electronic data processingsystem 328 determines, for a multitude of surface points X, acorresponding number of distances “d” between the laser scanner 300 andsurface points X on object 320. The distance to a particular surfacepoint X is determined based at least in part on the speed of light inair through which electromagnetic radiation propagates from the deviceto the surface point X. In one embodiment the phase shift between thelaser scanner 300 and the surface point X is determined and evaluated toobtain a measured distance “d.” In another embodiment, the elapsed time(the “time of flight” or TOF) between laser pulses is measured directlyto determine a measured distance “d.”

The speed of light in air depends on the properties of the air such asthe air temperature, barometric pressure, relative humidity, andconcentration of carbon dioxide. Such air properties influence the indexof refraction n of the air. The speed of light in air is equal to thespeed of light in vacuum “c” divided by the index of refraction. Inother words, c_(air)=c/n. A laser scanner 300 of the type discussedherein is based on the time-of-flight of the light in the air (theround-trip time for the light to travel from the device to the objectand back to the device). A method of measuring distance based on thetime-of-flight of light (or any type of electromagnetic radiation)depends on the speed of light in air.

In an embodiment, the scanning of the volume about the laser scanner 300takes place by rotating the rotary mirror 312 relatively quickly aboutthe axis 310 while rotating the measuring head 302 relatively slowlyabout the axis 306, thereby moving the assembly in a spiral pattern. Forsuch a scanning system, the gimbal point 308 defines the origin of thelocal stationary reference system. The base 304 rests in a localstationary frame of reference.

Referring to FIG. 4, an embodiment of a triangulation scanner 400 isshown that includes a light source 402 and at least one camera 404 andan electronic data processing system 420 that determines the 3Dcoordinates of points on the surface 410 of an object 408. Thetriangulation scanner may be similar to that described in commonly ownedU.S. patent application Ser. No. 14/139,021, filed on Dec. 23, 2013, thecontents of which are incorporated herein by reference. A triangulationscanner 400 is different than a laser tracker 200 or a TOF laser scanner300 in that the 3D coordinates of a surface of an object are determinedbased on triangulation principals related to the fixed geometricrelationship between the light source 402 and the camera 404 rather thanon the speed of light in air.

In general, there are two common types of triangulation scanners 400.The first type, sometimes referred to as a laser line probe or laserline scanner, projects a line or a swept point of light onto the surface410 of the object 408. The reflected laser light is captured by thecamera 404 and in some instances, the 3D coordinates of points on thesurface 410 may be determined. The second type, sometimes referred to asa structured light scanner, projects a two-dimensional pattern of lightor multiple patterns of light onto the surface 410 of the object 408.The 3D profile of the surface 410 affects the image of the patterncaptured by a photosensitive array within the camera 404. Usinginformation collected from one or more images of the pattern orpatterns, the electronic data processing system 420 can in someinstances determine a one-to-one correspondence between the pixels ofthe photosensitive array in camera 404 and the pattern of light emittedby the light source 402. Using this one-to-one correspondence togetherwith a baseline distance between the camera and the projector,triangulation principals are used by electronic data processing system420 to determine the 3D coordinates of points on the surface 410. Bymoving the triangulation scanner 400 relative to the surface 410, apoint cloud may be created of the entire object 408.

Further, there are generally two types of structured light patterns: acoded light pattern and an uncoded light pattern. As used herein, theterm coded light pattern refers to a pattern in which 3D coordinates ofan illuminated surface of the object are based on a single projectedpattern and a single corresponding image. With a coded light pattern,there is a way of establishing a one-to-one correspondence betweenpoints on the projected pattern and points on the received image basedon the pattern itself Because of this property, it is possible to obtainand register point cloud data while the projecting device is movingrelative to the object. One type of coded light pattern contains a setof elements (e.g., geometric shapes) arranged in lines where at leastthree of the elements are non-collinear. Such pattern elements arerecognizable because of their arrangement. In contrast, as used herein,the term uncoded structured light refers to a pattern that does notallow 3D coordinates to be determined based on a single pattern. Aseries of uncoded light patterns may be projected and imagedsequentially, with the relationship between the sequence of obtainedimages used to establish a one-to-one correspondence among projected andimaged points. For this embodiment, the triangulation scanner 400 isarranged in a fixed position relative to the object 408 until theone-to-one correspondence has been established.

It should be appreciated that the triangulation scanner 400 may useeither coded or uncoded structured light patterns. The structured lightpattern may include the patterns disclosed in the journal article“DLP-Based Structured Light 3D Imaging Technologies and Applications” byJason Geng, published in the Proceedings of SPIE, Vol. 7932, thecontents of which are incorporated herein by reference.

Collectively, the 3D metrology instruments such as the AACMM 100, thelaser tracker 200, the TOF laser scanner 300 and the triangulationscanner 400 are referred to herein as 3D instruments. It should beappreciated that these 3D metrology instruments are exemplary and theclaimed invention should not be so limited, as the systems and methodsdisclosed herein may be used with any 3D metrology instrument configuredto measure 3D coordinates of some object or scene.

Referring to FIG. 5, there illustrated is an augmented reality (AR)camera 500. Also illustrated is a 3D metrology instrument 510, such asthe aforementioned AACMM 100 of FIG. 1, the laser tracker 200 of FIG. 2,the TOF laser scanner 300 of FIG. 3, or the triangulation scanner 400 ofFIG. 4. Although not shown, other 3D metrology instruments may beutilized in embodiments of the present invention, including for examplea six degree of freedom (six-DOF) laser tracker used in combination witha six-DOF probe or a six-DOF scanner, and a triangulation scannerattached to a robot and having a position monitored by a camera bar. Itsuffices for the broadest scope of embodiments of the present inventionthat a 3D metrology or measurement device be provided that is used tomeasure the 3D coordinates of at least two cardinal points, themeasurement of the 3D coordinates of the at least two cardinal pointsbeing described in more detail hereinafter in conjunction withembodiments of the present invention.

An object 520 having a surface 524 is also shown in FIG. 5, whereinaccording to embodiments of the present invention, it is desired tocapture 2D images of some portion or all of the surface 524 of theobject 520 with the AR camera 500, and then process these 2D images suchthat an accurate or “true” 3D image of the surface 524 results.

The AR camera 500 is a 2D camera which may be considered to be one thatis capable of taking “full field” images. That is, the AR camera 500typically can obtain images over relatively large areas and of objectsat relatively large distances from the object 520. Also, the AR camera500 may provide enough information about the object surface 524 in itsimages to enable mapping of camera image data to a CAD model of theobject. The AR camera 500 may also provide color information about theobject surface 524. However, it should be understood that, according toembodiments of the present invention, the position and/or orientation(i.e., the “pose”) of the 2D AR camera 500 does not need to be knownwhen it obtains the 2D images of the object surface 524.

The AR camera 500 includes a camera lens 532 and a photosensitive array534. The photosensitive array 534 may be a CCD or CMOS array, forexample. Thus, the AR camera 500 may be digital in nature, and may takestill images or video images. The AR camera 500 may also include signalprocessing electronics 540 and memory 544. The signal processingelectronics 540 may allow for wireless or wired communication with the3D metrology instrument 510 and/or other devices (not shown) as neededin various applications of the AR camera 500. The AR camera 500 may alsoinclude one or more inertial sensors (not shown) which may assist inlocating the AR camera 500 within a particular frame of reference.

Within the lens 532 (which may be a lens system including a plurality oflens elements), there is a perspective center of the lens. The rays oflight passing through the lens 532 may be considered to pass through theperspective center before arriving at the photosensitive array 534. In acareful analysis, the lens 532 may be characterized to account for lensaberrations, which result in a slight shift in the intersectionpositions of the rays on the photosensitive array 534. However, withoutlosing generality, it is possible to say that the rays pass through theperspective center, with aberration correction to the image provided inanother step of image processing.

The surface 524 of an object 520 under investigation is imaged in twodimensions by the lens 532 onto the photosensitive array 534 to form anoverall 2D image on the collection of pixels that are a part of thephotosensitive array 534. Light falling on each pixel is converted,within an integration period of the camera, from a charge into a digitalsignal. An analog-to-digital converter, either located within thephotosensitive array 534 (for CMOS arrays) or external to the array 534(for CCD arrays), for example, as part of the signal processingelectronics 540, performs the conversion from an analog to a digitalsignal. The signal for each pixel is typically given in a binaryrepresentation of between 8 and 12 bits. The binary 1's and 0'srepresented by these bits are delivered over parallel channels, and maybe converted into serial form using a serializer/deserializer capabilityfor transmission over a bus line.

In embodiments of the present invention, the AR camera 500 is physicallyseparate from (i.e., not attached to) the 3D metrology instrument 510(or other measurement devices) such that the AR camera 500 is consideredto be a “stand-alone” device. The AR camera may be kept stationary byplacing it on a stationary mount, stand, or fixture; for example atripod. Mounting the AR camera 500 as such on a non-measurement devicedoes not change the status or nature of the AR camera 500 from that of a“stand-alone” device in the various embodiments of the presentinvention.

In embodiments of the present invention, multiple two-dimensional (2D)camera images taken by the AR camera 500 are combined or “registered”together according to a method, described hereinbelow, to obtain a“true” three-dimensional (3D) image representation of various real-worldfeatures such as, for example, a surface of an object or of somereal-world scene (e.g., the inside of a building, the location of avehicle accident, or a crime scene). The resulting relatively accurateor “true” 3D image of the 3D object surface 524 is obtained according toembodiments of the present invention by introducing a scale to anunscaled 3D image obtained from a step of matching cardinal pointsidentified within the 2D images obtained by the AR camera 500, where thescale is provided by distance measurements taken between at least twopoints (e.g., matching cardinal points) within the 2D images by the 3Dmetrology instrument 510.

A method according to exemplary embodiments of the present invention isnow described with reference to the method 600 shown in the flowchart ofFIG. 6. In a step 605, a 3D metrology instrument 510 is provided. Alsoprovided in the step 605 is a 2D stand-alone AR camera 500. Asmentioned, in embodiments of the present invention, the 3D metrologyinstrument 510 may, for example, be one of aforementioned AACMM 100 ofFIG. 1, the laser tracker 200 of FIG. 2, the TOF laser scanner 300 ofFIG. 3, or the triangulation scanner 400 of FIG. 4, each of which hasbeen described in detail hereinabove. Also, as mentioned hereinabove,the 3D metrology instrument or device 510 may comprise other types ofsimilar devices without departing from the broadest scope of embodimentsof the present invention.

A step 610 is, in a first instance, forming a first 2D image of the 3Dobject surface 524 with the stand-alone AR camera 500 disposed at afirst location (i.e., from a first observer perspective). The signalprocessing electronics 540 within the AR camera 500 may receive a firstdigital signal representing the first 2D image of the object surface 524sent through the camera lens 532 onto the photosensitive array 534 toform the first 2D image. Alternatively, the signal processingelectronics 540 may send the data from the photosensitive array 534 inthe form of the first digital signal to the 3D metrology instrument 510so that the instrument 510 may form the first 2D image. Also in thealternative, the signal processing electronics 540 may send the datafrom the photosensitive array 534 in the form of the first digitalsignal to some other signal processing device (not shown) which thenforms the first 2D image.

In a step 615, in a second instance, the stand-alone AR camera 500 ismoved to a second, different location (i.e., a second observerperspective different from the first observer perspective), and a second2D image of the object surface 524 is formed by the AR camera 500. Thesignal processing electronics 540 may receive a second digital signalrepresenting the second 2D image of the object surface 524 sent throughthe camera lens 532 onto the photosensitive array 534 to form the second2D image. Alternatively, the signal processing electronics 540 may sendthe data from the photosensitive array 534 in the form of the seconddigital signal to the 3D metrology instrument 510 so that the instrument510 may form the second 2D image. Also in the alternative, the signalprocessing electronics 540 may send the data from the photosensitivearray 534 in the form of the second digital signal to some other signalprocessing device (not shown) which then forms the second 2D image.

According to embodiments of the present invention, the first and second2D images should overlap to some extent such that there is some amountof imagery that is common to both the first and second 2D images. If theAR camera 500 is a color camera, then the first and second 2D images arecolor images.

In a step 620, at least three cardinal points in common to both thefirst 2D image and a second 2D image are found. The term “cardinalpoint” is typically used to refer to points that are identified as beingthe same point (i.e., “matching” points) in two or more of the 2Dimages. The cardinal points can then be used to connect or register theimages together. Also, cardinal points are typically not placedintentionally at their locations by someone. Further, the commoncardinal points may all be located on an object or within a scene.

There are a number of well-known techniques that may be used to findsuch cardinal points, generally using methods referred to as imageprocessing or feature detection. A commonly used but general categoryfor finding cardinal points is referred to as interest point detection,with the points detected referred to as interest points. According tothe usual definition, an interest point has a mathematicallywell-founded definition, a well-defined position in space, an imagestructure around the interest point that is rich in local informationcontent, and a variation in illumination level that is relatively stableover time. A particular example of an interest point is a corner point,which might be a point corresponding to an intersection of three planes,for example. Another example of signal processing that may be used isscale invariant feature transform (SIFT), which is a method well knownin the art and described in U.S. Pat. No. 6,711,293 to Lowe. Othercommon feature detection methods for finding cardinal points includeedge detection, blob detection, and ridge detection.

A step 625 is registering the first and second 2D images to obtain anunscaled, composite 3D image. Such an unscaled 3D image may be referredto as a “quasi-3D” image. This step of registering the two 2D images maybe based at least in part on the at least three cardinal points found instep 620, these at least three cardinal points being in common to boththe first and second 2D images.

Next, a step 630 is identifying (or selecting) a first cardinal pointand a second cardinal point from among the at least three cardinalpoints found in the step 620.

A step 635 is directly measuring the 3D coordinates of each of theidentified first and second cardinal points. In embodiments of thepresent invention, this step 635 is performed using the 3D metrologyinstrument 510, and is performed within a certain frame of reference(e.g., that of the 3D metrology instrument 510).

In an alternative embodiment, this step 635 may instead involve havingsome 3D coordinates obtained with the 3D measuring instrument 510provide the information needed to obtain the 3D coordinates of the atleast two cardinal points without directly measuring the 3D coordinatesof the cardinal points. For example, an object 520 may have two holesidentified by the AR camera 500 in at least two different 2D images.First and second cardinal points may then be established at the centersof each of the first and second circles, respectively, for each of twodifferent 2D images. By then measuring with a laser tracker 200 the 3Dcoordinates of the center of an SMR 204 (FIG. 2) placed on the holes,the distance between the centers of the circles may be determinedwithout directly measuring the cardinal points. The center of the firstcircle and the center of the second circle (i.e., the centers of theholes) may each be referred to respectively as a first reference pointand a second reference point, wherein the first reference point isassociated with or coincides with the first cardinal point and thesecond reference point is associated with or coincides with the secondcardinal point. The first and second reference points are with a regioncovered by both the first 2D image and the second 2D image. Thus, the 3Dcoordinates of the first and second reference points may be measuredwith the 3D metrology instrument 510.

Regardless of whether the 3D coordinates are measured directly by the 3Dinstrument 510 or are determined in some other manner, the result ofthis step 635 is that the 3D coordinates are used to provide thenecessary scaling to the unsealed 3D image that was created in the step625 described hereinabove from the first and second 2D images.

Finally, a step 640 involves determining a scaled composite 3D imagebased at least in part on the 3D coordinates of the first cardinal point(or of the first reference point), the 3D coordinates of the secondcardinal point (or of the second reference point), and the unsealedcomposite or “quasi” 3D image. Besides providing 3D coordinateinformation, a composite 3D image may also convey texture and colorinformation obtained not only from cardinal points but also from visibleregions between the cardinal points through use of, for example, knowninterpolation methods.

If the AR camera 500 is a color camera, the determined scaled composite3D image may illustrate the object surface 524 in color, and/or othersurface texture attributes may be within the composite 3D image.

Once the composite 3D images have been created by embodiments of themethod 600 of the present invention, these images may have data overlaidor superimposed thereon. For example, if the 3D images are those of anobject being built or already built, the data superimposed on the 3Dimages may comprise CAD design data of the object. The CAD data may bestored in memory associated with the laser tracker 200 (FIG. 2). Othertypes of data may be superimposed on the 3D images such as, for example,marks to indicate where various assembly operations (drilling,attaching, etc.) are to be performed.

Software may be used to observe the object and the surroundings fromdifferent perspectives and different distances, with the parallax shiftbetween the object and surroundings properly represented. In some cases,the background information may be important. For example, a project mayinvolve attaching a structure to the object being measured whileconfirming that there is adequate room in the 3D surroundings having a3D image obtained with the AR camera 500. Such a structure may beavailable as a CAD model, as a scanned image of a part or assembly, oras a scaled 3D representation obtained through the use of multiplecamera images.

In some cases, the AR camera 500 may be used to obtain representationsof areas ordinarily obstructed from view. For example, the AR camera 500may be used to view all sides of an object to obtain 3D images ofregions not easily measured otherwise. Such “full-view” coverage fromall directions of the object is particularly useful when images aredisplayed—for example, in a presentation, on a website, or in abrochure. The addition of color (texture) from the AR camera 500 is alsoof value in this instance. 3D representations obtained from the ARcamera 500 may be supplemented by other 3D representations. Models ofparts, assemblies, furniture, and so forth, may in some cases bedownloaded from files or websites and incorporated into a composite 3Drepresentation.

Another important use for the AR camera 500 and the 3D metrologyinstrument 510 is to obtain proper scaling of surroundings. For example,a wall may have a left side, a right side, an upper side, and a lowerside. Although the method of matching cardinal points describedhereinabove provides scaled 3D images, the dimensional accuracy willgenerally be much better if 3D coordinates are measured with the 3Dmetrology instrument 510 than with camera images alone. By combining thecomposite 3D image obtained from the 2D AR camera images with a fewmeasurements with the 3D metrology instrument 510, the scaling accuracyof the composite 3D image can, in many cases, be greatly improved. Forexample, improved scale of a building may be obtained by measuring oneor more positions on each of the left, right, upper, and lower sideswith the 3D metrology instrument 510.

The AR camera 500 may be used to measure only surroundings, onlyobjects, or both surroundings or objects. As the term is used here, theword “object” means an item for which accurate dimensional informationis desired. Measurement by an AR camera 500 provides the ability tosuperimpose full-view 3D images on drawings (for example, CAD) or other3D graphical models. In addition, by obtaining 2D images of an objectfrom multiple directions, it is possible to provide an overlay to anobject from all directions or sides of the object.

An object may be placed within its surroundings, the 3D coordinates ofwhich are obtained through the use of the AR camera 500. With theinformation provided by the AR camera and the 3D metrology instrument510, it is possible to view the objects from a variety of perspectivesrelative to its surroundings and also to view an object or itssurroundings from all directions.

In an embodiment, a purely graphical element (which could be aphotographic element, a drawn element, or a rendered element, forexample) is placed or superimposed within a composite image. A firstexample of such a graphical element is an addition to a machine tool ona factory floor within an interior building space. Such an addition maybe superimposed on a CAD model to which a composite color image isoverlaid. The addition might be a new machined part. A collection ofsuch additions may be placed in the context of a factory environment toensure that all elements fit properly. A second example of such agraphical element is a new item of machinery or furniture placed in thesame factory environment. A question might be whether such an elementwill fit in the new plans. In some cases, websites may be available thatenable downloading of such 3D images from the Cloud, which is a networktypically found on the Internet through a service provider. With someuser interfaces, such a 3D component may be moved into position with acomputer mouse and then viewed from different positions andorientations.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

While the invention has been described in detail in connection with onlya limited number of embodiments, it should be readily understood thatthe invention is not limited to such disclosed embodiments. Rather, theinvention can be modified to incorporate any number of variations,alterations, substitutions or equivalent arrangements not heretoforedescribed, but which are commensurate with the spirit and scope of theinvention. Additionally, while various embodiments of the invention havebeen described, it is to be understood that aspects of the invention mayinclude only some of the described embodiments. Accordingly, theinvention is not to be seen as limited by the foregoing description, butis only limited by the scope of the appended claims. Moreover, the useof the terms first, second, etc. do not denote any order or importance,but rather the terms first, second, etc. are used to distinguish oneelement from another. Furthermore, the use of the terms a, an, etc. donot denote a limitation of quantity, but rather denote the presence ofat least one of the referenced item.

What is claimed is:
 1. A method of combining at least twotwo-dimensional (2D) images into a three-dimensional (3D) image, themethod comprising steps of: providing an augmented reality (AR) camera;providing a 3D metrology instrument having a frame of reference, the ARcamera being separate from the 3D metrology instrument, wherein the 3Dmetrology instrument is selected from the group comprising a lasertracker, a time-of-flight (TOF) laser scanner, and an articulated armcoordinate measurement machine; in a first instance, forming a first 2Dimage with the AR camera in a first location; in a second instance,moving the AR camera to a second location that is different from thefirst location, and forming a second 2D image with the AR camera;finding at least three cardinal points in common to both the first andsecond 2D images; registering the first and second 2D images to obtainan unscaled composite 3D image based at least in part on the at leastthree found cardinal points; identifying a first selected cardinal pointand a second selected cardinal point from among the at least three foundcardinal points; measuring 3D coordinates of a first reference point anda second reference point with the 3D metrology instrument, the firstreference point and the second reference point being within a regioncovered by both the first 2D image and the second 2D image; determining3D coordinates of each of the first and second selected cardinal pointsbased at least in part on the measured 3D coordinates of the firstreference point and the second reference point; and creating the 3Dimage as a scaled composite 3D image from the determined 3D coordinatesof each of the first and second selected cardinal points, and from theunscaled composite 3D image.
 2. The method of claim 1 wherein, in thestep of measuring 3D coordinates of a first reference point and a secondreference point with the 3D metrology instrument, the first referencepoint is associated with the first selected cardinal point and thesecond reference point is associated with the second selected cardinalpoint.
 3. The method of claim 2 wherein, in the step of measuring 3Dcoordinates of a first reference point and a second reference point withthe 3D metrology instrument, the first reference point coincides withthe first selected cardinal point and the second reference pointcoincides with the second selected cardinal point.
 4. The method ofclaim 2 wherein, in the step of measuring 3D coordinates of a firstreference point and a second reference point with the 3D metrologyinstrument, the first reference point is the center of a hole.
 5. Themethod of claim 1 further comprising superimposing data on the 3D image.6. The method of claim 5 wherein, in the step of superimposing data onthe 3D image, the superimposed data includes computer-aided design (CAD)data.
 7. The method of claim 5 wherein, in the step of superimposingdata on the 3D image, the superimposed data includes marks indicative ofan assembly operation.
 8. The method of claim 5 wherein, in the step ofsuperimposing data on the 3D image, the superimposed data is a graphicalelement selected from the group consisting of a photographic element, adrawn element, and a rendered element.
 9. The method of claim 1 furtherincluding providing a first 2D representation of the 3D image, the first2D representation obtained from a first observer perspective.
 10. Themethod of claim 9 further including providing a second 2D representationof the 3D image, the second 2D representation obtained from a secondobserver perspective different than the first observer perspective. 11.The method of claim 1 wherein in the step of finding at least threecardinal points in common to both the first and second 2D images, thecommon cardinal points are on an object.
 12. The method of claim 11further including a step of obtaining images from all sides of an objectand, in response, determining a full-view 3D image, the full-view 3Dimage being a 3D image of all sides of the object.
 13. The method ofclaim 12 further including superimposing the full-view 3D image onto a3D graphical model.
 14. The method of claim 13 wherein the 3D graphicalmodel is a computer-aided design (CAD) model.
 15. The method of claim 13wherein the 3D graphical model is a 3D model of an interior buildingspace.
 16. The method of claim 1 wherein: in the step of providing anaugmented reality (AR) camera, the AR camera is a color camera; in thestep of forming a first 2D image with the AR camera in a first location,the first 2D image is a color image; in the step of moving the AR camerato a second location that is different from the first location, andforming a second 2D image with the AR camera, the second 2D image is acolor image; and in the step of creating the 3D image, the 3D image is acolor image based at least in part on the first 2D image and the second2D image.